Distribution by Document Size

Search engines split large datasets across multiple machines using document distribution. Documents are typically distributed randomly to produce good load balancing. We propose that documents be distributed by their size instead. This can make load balancing more difficult, but it produces immediate improvements in both index size and query throughput. To support our proposal, we show improvements to an in-memory conjunctive list intersection system running on the GOV2 dataset using simple16 compression combined with either skips or bitvectors. While our list intersection system does not implement ranking, we also expect significant performance improvements from using document size distribution in ranking based search systems. In addition, implementations can be adapted to produce further performance improvements that exploit the document size distribution. We present some examples that apply to a full ranking based search engine and leave their verification for future work.